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This is one of a series of reports based on an 
ongoing reality test of systemic evaluation for instructional 
decision making. This feasibility study is being carried out by the 
Center for the Study of Evaluation with the Laboratory in School and 
Community Relations at a suburban Los Angeles high school (called 
Site A). Viewing a school as a cultural/ecological system, systemic 
evaluation is a set of principles to guide local development of 
processes to gather, organize, and utilize information relevant to 
the needs and values of the system and its members. This paper 
discusses the potential abuses of comprehensive, school-oriented 
information systems. There are at least three issues that must be 
addressed if the systems are to be used by and for people instead of 
against them: (1) intimately involving people in developing the 
purposes, contents and uses of the systems; (2) developing timely, 
relevant, and straightforward techniques for bringing information to 
bear upon practice; and, (3) creating a climate of mutual trust and 
responsibility for generating and using information. (BS) 
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For the past dozen years, I have spent most of my professional work 
life thinking about, writing about, and using comprehensive data bases on 
schools and schooling. About two thirds of this time was spent working 
with John Goodlad and my other colleagues on A Study of Schooling— a study 
that attempted to develop a comprehensive, data-based description of what 
goes on in schools and how people feel about it— all in an effort to draw 
some working hypotheses about how things might happen differently (Goodlad, 
1983). More recently, a few colleagues and I at the Center for the Study 
of Evaluation and at the Laboratory in School and Community Education have 
paused to reflect upon the role of comprehensive data-bases, less from the 
Standpoint of doing a research study across schools, and more with an eye 
towards how information can be used for local school improvement and 
change.^ 

I note this brief, personal history only to support my commitment to 
the whole idea of comprehensive, school -oriented information systems. Yet 
as excited as I am about the potential use of the idea, I am worried about 
the potential of its abuse . It seems increasingly clear to me how people 
can use school information systems* And it seems increasingly clear how 
such systems can use people. If the latter turns out to be more the rule 
than the exception, I will be looking to become involved in a different 
line of educational inquiry. 

To be more specific about my worries, I would like to divide the 
subsequent remarks into three categories: 1. inquiry and the role of 
information, 2. humanizing data, and 3. accountability vs. responsibility. 



The first category is really preeminent with the latter two categories 
being corollaries of the position being taken on information systems. 



Inquiry and the Role of Information 

At the outset, it is important to clarify the perspective underlying 
the kind of "information system" that I and my colleagues have envisioned. 
We would not characterize it as a district-driven, "management information 
system" exclusively (or nearly exclusively) oriented around monitoring 
achievement, i.e., norm- and/or criterion-referenced test score results. 
Rather, what we have in mind is more of a school-based, inquiry-driven 
system of information to be collected and used formatively in the process 
of school improvement. 

It is virtually impossible to do justice to the ideas embedded in this 
view of information system in this short report. My hope is that those of 
you interested in these ideas well look at the reports referenced in this 
paper. All I can do here is summarize in shot-gun fashion some core 
concepts: 

1. Outcome indices have limited value, beyond their immediate, 
descriptive signal, for helping direct an agenda for school 
improvement. Certainly I am not suggesting that assessing student 
achievement is unnecessary. Rather, I am suggesting that test 
scores alone are an insufficient basis for action. 

2. What is further necessary is relevant information on the 
circumstances, activities and meanings associated with the 
schooling process. By "circumstances," I mean the whole array of 
structures, situations, and physical features in the school 
(condition of building, teacher-student ratio, student tracking 
policy, materials and resources, and so forth). By "activities," 

I mean the behaviors and processes that constitute the practice of 
schooling (staff decision-making, classroom teaching strategies, 
student socialization patterns and so forth). By "meanings," I 
mean the understandings that people infer from and bring to bear 
upon schooling (orientations like feelings, opinions, attitudes, 
beliefs and values are included here as well as any other 
indicators of meaning—test scores for example). 
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3^ The criteria for relevance are based upon the perceived needs of 
the significant "actors" in the setting (e.g., administrators, 
teachers, students, parents) and the inherent value systems 
through which these perceptions are filtered. 

4. Information systems, as knowledge producing and using processes , 
must reflect the variety m: 

a. types of information, e.g., cognitive, affective, contextual. 

b. commonplaces (Goodlad, 1979) of schooling, e.g., physical 
environment, human resources, material resources, organization, 
communication, problem-solving/decision-making, leadership, " 
issues/problems, controls/restraints, expectations, climate, 

and curriculum. Curriculum is to be interpreted broadly and 
should include at least these additional commonplaces: 
goals/objectives (academic, personal, social, career), content, 
instructional materials, classroom activities, teaching 
strategies, assessment of student learning, use of time and 
space, and grouping strategies. 

c. domains of information, e.g., individual, class, team/grade/ 
department, school, district, community. 

d. sources of information, e.g., students, teachers, administrators, 
parents, observers, documents. 

e. methods for collecting information, e.g., tests, surveys, 
observations, interviews, case studies, historical analyses, 
document reviews. 

I have tried to pack most of these features into figuT'e 1 which is offered 
as a heuristic for suggesting the depth and breath of potential school 
information. (Note also that Figure 1 represents just one slice out of 
many possible along the pre-school , K-12, post-secondary continuum.) 

With these concepts as a backdrop, I can now be more explicit regarding 
the theme of this paper: people using or being used by information systems. 
There is a fifth and most important concept to be added to the above list: 

5. Information systems, as knowledge producing and using processes, 
must be multi-paradigmatic, embracing the tensions between these 
alternative inquiry perspectives: 

a. Empirical analytic methods that place a prem"' m on explanation 
via preaictive relationships between quantified constructs. 

b. Natural istic/phenomenological methods that place a premium on 
understanding via qualitative interpretations of social settings. 
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Critical/dialecti cal methods that place a premium on informed 
discourse, values'clarification, and action* 

Thus, a district or school seriously bent upon sustained improvement 
and change efforts will need to involve school staff in the collaborative 
pursuit of understanding— What goes on in their school (s)? How did it come 
to be that way? What are the social, political and economic interests that 
constrain the setting? Reconciling the various phenomenological views of 
the setting and approaching a working consensus on problem areas should 
always the first order of business. As the dialogue proceeds, it will 
become evident that much information is needed--1nfon!iation that can 
be determined through various operational devices (e.g., surveys, tests, 
observations) or information that is already available (e.g., school 
records) but needs to be organized and disseminated. Only when Information 
is perceived as useful, should Information systems be conceived for use. 

To summarize, information as knowledge is not an end in Itself but is, 
instead, a catalyst for evaluative discourse and action; inquiry and 
information need to be legitimized as natural and on-going parts of the 
daily work life of those for whom the knowledge is to be relevant. 

Humanizing Dat a 

Many professional and lay persons both Inside and outside of the 
educational research and schooling communities have never been enamored 
with the notion of quantifying the meaning of circumstances and events in 
social settings. To exacerbate matters further, the exponential rise of 
high technology has propelled us into an *'age of information." The only 
way to escape being "computerized" is to disenfranchise oneself from 
economic life. Ky guess is that these societal changes, coupled with past 
sentiments regarding "research-type" activities, will make those people 



identified as potential data sources even less sanguine— and more cynical 

and suspicious—regarding the benefits of the kind of comprehensive information 

systems being described. 

Picking up on the theme from the previous section, a crucial ingredient 
required to overcome this concern is the cultivation of an attitude towards 
information that makes it an intrinsic part of an organizational environment 
that legitimizes pr ofessional inquiry by allocating quality time^ to_ the 
effort. But there are a few other, more mundane ingredients having to do 
with attitudes toward and techniques of measurement and data analysis * 

Perhaps the most important problem in measurement is to overcome two 
kinds of attitudes that tend to polarize people into either of two belief 
"camps," described by Kaplan (1964) as adherents of either the "mystique of 
quantity" or the "mystique of quality." The extreme position in the former 
camp is embodied in the expression, *'If you can't measure it, that ain't 
it." And the opposite extreme in the latter camp— "If you can measure it, 
that ain't it." 

As with all false dichotomies, the truth is somewhere in between and 
is rooted pragmatism. It is unreasonable to believe that the mathematical 
power inherent in numbers somehow transcends the strength (or weakness) of 
their connections with properties they are presumably measuring. It is 
equally unreasonable to assume that numbers assigned to reified concepts 
(such as "self-esteem" and "principal leadership") cannot possibly represent 
anything meaningful. 

The ultimate arbitrator of the meaning of measurement is experience. 
This is why ths notions of reliability and validity were invented. To the 
extent that the numbers (i.e., measurements) can be replicated, they are 
reliable. More importantly, to the extent that they serve the measurement 



purposes intended, they are valid. The key word here is purpose . 
Depending upon the purpose, the evidential arguments for reliability and 
validity may differ. 

Consistent with the primary objective being espoused here for collecting 
and using data—to inform and stimulate staff inquiry— evidence must be 
acquired first for the content validity and second for the credibility of 
information. Content validity is a familiar concept to most people who 
construct achievement tests. It is essentially a rational process of 
matching item content to instructional content, i.e., the course (or unit 
or lesson) objectives and the skills, knowledge, and understandings implied 
by those objectives. Likewise, the items in surveys, interviews and 
observation schedules must be matched to the content they are designed to 
assess. A concept such as "staff cohesiveness" may be of important concern 
to the organizational work environment in a school. But within the context 
of that school, a concept like "staff cohesiveness" needs to be scrutinized 
for its various meanings. Does it represent support? friendliness? 
trust? respect? morale? commitment? unity? etc.? 

The term "credibility" as a type of validity is used here to represent 
the degree to which information augments, stimulates, provokes or otherwise 
facilitates meaningful communication between staff in the inquiry process. 
Assessing credibility can only be done by the staff during thB coursa of 
inquiry. Suppose the issue of increasing parent involvement in school 
affairs is under deliberation. Suppose the results of a parent survey 
question regarding the reasons they have for feeling disenfranchised from 
school affairs are added to the information being brought to bear on the 
inquiry. If the results lead to a "so what?" response or an inability to 
relate the data to the pertinent issue, it us unlikely that the item will 



be included in further surveys. On the other hand, if the results can be 
seen to further the dialogue, the item has proved credible. (See example 
below for Nuvo Elementary School). The point here is that issues pertaining 
to the validity of information are not exclusively those facing psychometri- 
cians. These issues must be continually addressed as people attempt to 
make sense out of the information collected. 

This leads, then, to the kinds of data analyses likely to be of use to 
staff for facilitating their inquiry about the conditions of their schools 
and the possible avenues for improvement and change. It is unlikely that 
the results of multivariate analyses such as multiple regression, covariance 
structure modelling, and the like, will be of use in this effort. Obviously, 
such analyses can be useful for research purposes in studying complex 
relationships between variables and how they change over time. Y6t, they 
lack the immediacy and simplicity required to inform ,taff deliberation on 
specific problems and issues. 

But do not confuse simplicity with simple-mindedness. The myth that 

complex numerical manipulations somehow yield better, more "scientific" 

results has no place in the rationale for a "people-oriented" information 

system. The power of a single percentage should n^.\, be underestimated as a 

stimulus for facilitating and advancing the dialogue. A brief summary of 

the events of a staff meeting at one elementary school should illustrate 

how simple tabulations of data can facilitate staff inquiry. 

A continuing issue at Nuvo Elementary School concerned curriculum 
balance and the role of content area specialists. Prior to this 
meeting it had been suggested that staff really didn't know how much 
time was being devoted to various subject areas in each grade level. 
As an approximation to this bit of missing knowledge, staff responded 
to a question asking for the approximate, weekly number of hours 
allocated to a number of subject area divisions. Since teachers at 
this school taught in 10 teams (of 2-3 teachers each) spread across 
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grade levels, teams (rather then individuals) reached consensus on 
this item; and the 10 team responses were arrayed and presented as 
input to the staff meeting. 

Preliminary discussion began around the nature of the item itself and 
the difficulty of cutting up the hours of the day to correspond to the 
subject matter categories. Thus, to some extent, the hours indicated 
by teams were not realistic. Yet, all teams felt that the general 
patterns in the data "rang true." These patterns were two-fold: 
(1) There were extreme imbalances in the time allocated to different 
content areas and (2) The nature of those imbalances were very different 
in different grade levels and teams. These observations fed back 
nicely into the major thrusts of the issue. First, what ought be the 
curricular balance between subject contents, should it be different at 
different grade levels, and, if so, how can balance be maintained in 
the continuum from one grade level to the next? 

But the original criticism of the survey question really highlighted a 
second thrust. How separable are content areas, and to what extent do 
we (and should we) teach subjects (e.g., reading, math and science) 
together as they naturally occur within a thematic unit (e.g., ecology)? 
This query, of course, raised the role of content specialists as being 
"outside class" resources versus being regular members of a_ team with 
special talents that can be shared with other staff as needed. 

This is enough of a scenario to make the point regarding how simple i 

(not simplistic) survey results can facilitate inquiry. It should also be ' 

noted that content validity and credibility issues were implicit in this 

scenario and could be made explicit during the course of the inquiry. 



Responsibility vs. Accountability 

Teachers, students, parents, etc. have been "burned" far too often by 
mindless exercises of data collection (usually surveys), the results of 
which never see the light of day or, if they do, are presented in a useless 
form, in a useless setting, and/or at a useless time. Even worse, however, 
are the scars left from using information as a weapon, often under the 
rubric of accountability . Host often, the arsenal is composed of test 
score information, but I have verified "war stories" where other types of 
information (e.g., teacher educational values/beliefs; instructional 
practijes; etc.) have been used against people (e.g., RIF decisions). 
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I find this attitude towards and use of information to be intolerable^ 
It is not surprising, then, that it is wholly incompatible with the view 
being espoused here of information systems as an adjunct to and by-product 
of a school -focused inquiry and change process. From this perspective, 
"evaluation is no longer a tagged-on prerequisite; it is intrinsic to the 
reflective process and is engaged in by all involved. "Accountability" is 
recast to a higher ethical level; people become accountable to one another 
rather than to what are often seen as arbitrary, outside interests. 
Responsibility would seem to be a more reasonable term to characterize this 
view of inquiry and the use of information. 

Summary 

There are at least three issues that must be taken seriously by 
information system advocates if the systems are to be used by and for 
people instead of against them: 

1. Intimately involving people in developing the purposes, contents 
and uses of the systems, 

2. Developing timely, relevant and straightforward techniques for 
bringing information to bear upon practice. 

3. Creating a climate of mutual trust and responsibility for 
generating and using information. 



ERIC 



10 



10 



Footnote 



!• The fcllowing reports provide a more thorough discussion of all the 
concepts embedded in this paper: Sirotnik, 1984; Sirotnik and 
Burstein, 1983; Sirotnik and Oakes, 1981 and 1983. 
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